A Comparison of Two Models for Cognitive Diagnosis

Diagnostic score reports linking assessment outcomes to instructional interventions are one of the most requested features of assessment products. There is a body of interesting work done in the last 20 years including Tatsuoka’s rule space method (Tatsuoka, 1983), Haertal and Wiley’s binary skills model (Haertal, 1984; Haertal & Wiley, 1993), and Mislevy, Almond, Yan, and Steinberg’s Bayesian inference network (Mislevy, Almond, Yan, & Steinberg, 1999). Recent research has resulted in major breakthroughs for the use of a parametric IRT model for performing skills diagnoses. Hartz, Roussos, and Stout (2002) have developed an identifiable and flexible model called the fusion model. This paper compares the fusion model and the Bayesian inference network in the design and analysis of an assessment, including the Q-matrix, and then compares other related models, including rule space, item response theory (IRT), and general multivariate latent class models. The paper also attempts to characterize the kinds of problems for which each type of measurement model is well-suited. A general Bayesian psychometric framework provides common language, making it easier to appreciate the differences. In addition, this paper explores some of the strengths and weaknesses of each approach based on a comparative analysis of a cognitive assessment, the mixed number subtraction data set. In this case, both the fusion model and Bayesian network approaches yield similar performance characteristics and also seem to pick up on different characteristics.